Video and Clip Examples

This example will show how to use ApertureDB's Video and Clip functionality.

We will start by connecting to an ApertureDB instance that has Videos and Clips:

from aperturedb import Connector

db = Connector.Connector("videos.develop-gcp.aperturedata.io", user="admin", password="admin")

We will start by adding a new video, from the YFCC100m dataset:

!wget https://multimedia-commons.s3-us-west-2.amazonaws.com/data/videos/mp4/1f8/330/1f83301f5bfcd37f5ecdc71ae20bffe.mp4

query = [{
    "AddVideo": {
        "properties": {
            "source": "yfcc100m_dataset",
            "label": "nature",
            "guid": "1f83301f5bfcd37f5ecdc71ae20bffe"
        },
        "if_not_found": {
            "guid": ["==", "1f83301f5bfcd37f5ecdc71ae20bffe"]
        }
    }
}]

fd = open("1f83301f5bfcd37f5ecdc71ae20bffe.mp4", 'rb')
video_blob = fd.read()

response, blobs = db.query(query, [video_blob])
db.print_last_response()

[
    {
        "AddVideo": {
            "status": 0
        }
    }
]

Now that the video was inserted successfully, we will retrieve it using the "guid" property as a filter:

query = [{
    "FindVideo": {
        "blobs": True,
        "constraints": {
            "guid": ["==", "1f83301f5bfcd37f5ecdc71ae20bffe"]
        },
        "results": {
            "all_properties": True
        }
    }
}]

response, blobs = db.query(query)
db.print_last_response()

[
    {
        "FindVideo": {
            "blobs_start": 0,
            "entities": [
                {
                    "_blob_index": 0,
                    "_duration_us": 27427400,
                    "_fps": 29.97002997002997,
                    "_frame_count": 822,
                    "_frame_height": 360,
                    "_frame_width": 640,
                    "_uniqueid": "4.1000.103000",
                    "guid": "1f83301f5bfcd37f5ecdc71ae20bffe",
                    "label": "nature",
                    "source": "yfcc100m_dataset"
                }
            ],
            "returned": 1,
            "status": 0
        }
    }
]

We can see that ApertureDB computes some system-defined properties from the videos, that can be used for filtering later.

We can use ApertureDB NotebookHelpers to display the retrieved videos:

from aperturedb import NotebookHelpers as nh

nh.display_video_mp4(blobs[0])

We can see that the video has 2 parts: a part that shows birds in the sky, and a second part that shows birds on the pond.

For this example, let's say we are interested in the second part of the video, because we want to observe only the birds on the pond. For this, we can use ApertureDB Clip capabilities to define and extract only the last portion of the video.

We will use "time_fraction_offset" to indicate the clip should only contain the last 50% of the video:

query = [{
    "FindVideo": {
        "_ref": 1,
        "blobs": False,
        "constraints": {
            "guid": ["==", "1f83301f5bfcd37f5ecdc71ae20bffe"]
        },
    }
}, {
    "AddClip": {
        "video_ref": 1,
        "time_fraction_range": {
            "start": 0.5,
            "stop": 1.0
        },
        "properties": {
            "label": "birds_on_pond"
        }
    }
}]

response, blobs = db.query(query)
db.print_last_response()

[
    {
        "FindVideo": {
            "returned": 0,
            "status": 0
        }
    },
    {
        "AddClip": {
            "status": 0
        }
    }
]

We can see that the clip was correctly added. Note that a Clip in ApertureDB stores only the position of the clip within the video as metadata properties. There is only one copy of the video in ApertureDB.

Now, let's retrieve the recently created Clip:

query = [{
    "FindClip": {
        "blobs": True,
        "constraints": {
            "label": ["==", "birds_on_pond"]
        },
        "fast_cut": True,
        "results": {
            "all_properties": True
        }
    }
}]

response, blobs = db.query(query)

db.print_last_response()

[
    {
        "FindClip": {
            "blobs_start": 0,
            "entities": [
                {
                    "_range_start": 50000,
                    "_range_stop": 100000,
                    "_range_type": 2,
                    "_time_fraction_range": {
                        "start": 0.5,
                        "stop": 1.0
                    },
                    "_uniqueid": "5.3000.103440",
                    "label": "birds_on_pond"
                }
            ],
            "returned": 1,
            "status": 0
        }
    }
]

ApertureDB extracts the clip from the original video on the fly.

Let's now display the clip:

from aperturedb import NotebookHelpers as nh

nh.display_video_mp4(blobs[0])

We can also apply on-the-fly transformations to the video, for example we can resize the video to 320x240 pixels:

query = [{
    "FindClip": {
        "blobs": True,
        "constraints": {
            "label": ["==", "birds_on_pond"]
        },
        "fast_cut": True,
        "operations": [
            {
                "type": "resize",
                "width": 320,
                "height": 240
            }
        ],
        "results": {
            "all_properties": True
        }
    }
}]

response, blobs = db.query(query)

db.print_last_response()

nh.display_video_mp4(blobs[0])

[
    {
        "FindClip": {
            "blobs_start": 0,
            "entities": [
                {
                    "_range_start": 50000,
                    "_range_stop": 100000,
                    "_range_type": 2,
                    "_time_fraction_range": {
                        "start": 0.5,
                        "stop": 1.0
                    },
                    "_uniqueid": "5.3000.103440",
                    "label": "birds_on_pond"
                }
            ],
            "returned": 1,
            "status": 0
        }
    }
]